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Program  Trends  and  Strategy: 

TOPIC  AREA  OVERVIEW 


Advanced  Auditory  Modeling: 


Scientific  Question:  How  does  the  auditory  brain  parse  acoustic 
landscapes,  bind  sensory  inputs,  adapt  its  filters,  hear  through  noise  and 
distortion?  Could  autonomous  listening  devices  emulate  neurology  to  match  or 
exceed  human  auditory  analysis,  e.g.,  to  detect  and  identify  speech  targets  in 
noise  and  reverberation? 


Polarization  Vision  &  Optics: 


12% 


Scientific  Question:  How  do  natural  photoreceptors  detect  and  how  do 
animal  brains  interpret  polarization  information?  How  is  it  used  for  nocturnal 
navigation  or  recognition  of  obscured  targets?  Can  these  unique  bio-optical 
systems  be  emulated?  AFOSR 

4y0/o  BioNaviqation 

Sensorimotor  Control  of  Flight  &  Navigation:  initiative 

Scientific  Question:  How  does  neural  control  make  natural,  low-Reynolds 
No.  flight  autonomous,  efficient,  and  robust?  Discover  principles  of 
multisensory  fusion,  distributed  sensors  and  actuators.  Develop  control  laws 
for  emulation  in  MAVs. 


Strategy:  Forge  useful  connections  between  math  and  biology 
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3 


Grantees’  Self-Organized 
Research  Workshops  (2011) 


Neural  Oscillations  and  Speech  Perception. 

24  May.  Hosted  by  NYU. 


Science  &  Applications  for  Bio-Inspired  Navigation  and  Control. 


23-25  May.  Hosted  by  AFRL/RW. 

Bio-Optics  and  Photoreception. 

27-30  June.  Hosted  by  U.  Bristol. 

Funded  by  the  Rank  Prize  Foundation 


Encoding  for  Auditory  Representations. 

22-23  August.  Hosted  by  U.  Washington. 

Informational  Masking  &  Binaural  Hearing. 

17-19  Nov.  Hosted  by  Boston  U. 


Convened 
by  grantees; 
No AFOSR 
workshop 
funding 


Brain  Rhythms  and  Cortical  Computation. 

18  Nov.  Hosted  by  NYU. 
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V  J  Sensory  Information  Systems 

Program  --  People 


Dr.  Jennifer  Talley, 
Biology 

Dr.  Talley  (Ph.D.  2010)  was 
recruited  from  AFOSR- 
sponsored 

project  at  Case  Western  U. 
to  insect  flight  lab  at 
AFRL/Eglin 


1  ^  Dr.  Eric  Thompson, 

Judith  A.  Resnik  Medal  Acoustics 

Dr.  Pramod  Varshney, 

Syracuse  U.  Dr.  Thompson  (Ph.D.  2009) 

was  recruited  from 
AFOSR-sponsored 
projects  at  Boston  Univ.  to 
psychoacoustics  lab  at 
AFRL/RH 
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RECENT  HIGHLIGHTS 


OPSIN  MAP 


New  Genomic 
Analysis  reveals 
maximum-likelihood 
phylogenetic  model 
for 

Photoreceptor  Opsins 


AFOSR 

Polarization  Biology 
Sub-program 
featured  in 
J.  Royal  Society 
Special  Issue 


DR.  KEY  DISMUKES 


WRIGHT  STATE 
UNIVERSITY 


Keynote  Speaker  for 
AFRL  -  Wright  State 
Internat.  Symposium 
on 

Aviation  Psychology 
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Recent  Transitions 


TO:  Navy:  Bio-inspired  method  to  classify  acoustic  sources,  e.g.,  vehicles, 
humans,  marine  animals  etc.,  using  cortical  auditory  model.  Dr.  Sam  Pascarelle, 
Advanced  Acoustic  Concepts,  Inc. 

TO:  AFRL--  Eglin:  Measurements  and  data  on  biological  wide-field-of-view 
optical  systems  to  enable  6.2  and  6.3  efforts  in  vision-based  guidance  and 
navigation.  D.  Stavenga,  U.  Groningen,  N.  Strausfeld,  U.  Arizona,  M.  Wehling,  et  al. 


TO:  Bloedel  Hearing  Institute:  New,  patented  method  to  improve 
auditory  coding  in  cochlear  implants.  Developed  by  Dr.  Les  Atlas,  U.  Wash.  Dr.  Jay 
Rebenstein  will  develop  commercial  applications. 

TO:  AFIT:  Techniques  in  electrophysiology  and  neuroanatomy  for 
mechanical  engineering  projects  to  emulate  flapping  wing  flight.  Dr.  Mark 
Willis,  Case  Western  U.,  Dr.  Anthony  Palazotto,  AFIT 


TO:  DARPA:  System  for  adaptive,  autonomous  control  of  robotic 
movement,  based  upon  hierarchical  neural  model  of  biological  control. 
Roger  Quinn  &.  Roy  Ritzmann,  Case  West.  U.;  G.  Pratt,  DARPA. 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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Autonomous  Steering: 

Transition  to  Army  MAST 


Quadrotor  implements 
autonomous  3D  navigation  using 
wide  field-of-view  optic  flow 


Inspiration: 

•  Insect  Compound  Eye  Research 

Eglin  AFB  6. 1  (M.  Wehling,  et  al.) 

•  Insect  Visual  Target  Detection  Modeling 

Tanner  Research  (P.  Shoemaker.) 

U.  Adelaide  (D.  O’Carroll) 

•  Insect  Flight  Behavior  Modeling 

U.  MD.  (J.  S.  Humbert) 

Microautonomous  Systems  and  Technology 
http  ://www.  avl.umd.edu/ 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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Bio-Emulation  Progress 

From  6.1  at  U.  Queensland,  U.  Bristol,  &  UMBC 


1.  Discovery: 

Polarization  photoreceptor  imparts 
^-invariant  90°  phase  delay 


Honeycomb  structure  of 
photoreceptor’s  optical 
retarder  membrane 


2.  Optical  Modeling: 


“Nature’s  Perfect  Waveplate” 


Key  property: 

Intrinsic  birefringence  of  microvilli 
elements,  in  balance  with  form 
birefringence,  yields  wavelength- 
invariant  phase  delay. 


3.  Emulation 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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Nano-Rod  Deposition  Recipe  Emulates 
Biological  Achromatic  Phase  Retarder 
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All  4  Stokes 
parameters  remain 
nearly  constant  in  a 
multi-layer  structure 


Yi-Jun  Jen,  Nature  Comm.  21  June  2011 


Tilted  nanorods 
(yellow  layers) 
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V  J  Auditory  Modeling  for  Acoustic 

Analysis  —  Extramural  Grants 


L.  Atlas  (U.  Washington): 

New  approach  to  modulation  representation  and  filtering. 


E.  Bleszynski  (Monopole  Research): 

Math  model  of  bone-  &  tissue-conducted  sound 

M.  Elhilali  (Johns  Hopkins  U.): 

Cortical  model  for  acoustic  scene  segregation 

O.  Ghitza  (Boston  U.): 

Theory  of  speech  parsing  via  brain  rhythms 

W.  Hartmann  (Mich.  State  U.): 

Sound  localization 

G.  Kidd  (Boston  U.): 

Informational  masking  &  speech  segregation 


Mathematical  and 
Psychoacoustic 
Research  to 
Understand  &  Emulate 
the  Human  Listener's 
Auditory  Capabilities 
and  to  Enhance  these 
Capabilities  where 
Possible . 


R.  Kumaresan  &  P.  Cariani  (U.  Rhode  Is.  &  Harvard): 
Spectro-temporal  codes  for  auditory  signal  representation 

D.  Wang  (Ohio  State  U.): 

Computational  auditory  scene  analysis 

DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 


4  NEW  TECHNICAL 


New  Technique  Isolates 
Informational  &  Energetic  Masking 


Background: 

•  What  listeners  can  hear  in  real, 
multisource  acoustic  environments  is 
more  constrained  by  informational 
masking,  than  by  direct,  energetic 
interference  of  one  sound  by  another. 

•  AFOSR  seeks  techniques  to  study 
&  suppress  informational  masking. 

•  Sparse  pointillistic  coding  leaves 
speech  remarkably  intelligible, 
eliminates  energy  overlaps,  isolates 
effects  of  informational  “cross-talk.” 


A  sparse,  pointillistic  rendition  of 
the  word  “SHOES,”  overlaid  on  the 
conventional  spectrogram 


0  1/  0.2  0.3  0.4  0.5  0.0 


I  ime  (sec) 


Each  time/frequency  slice  of  speech  has  been 
replaced  with  a  matching  cosine  fragment. 


•  Briefed  to  AFRL/RI  in  Dec.  2011 . 


Red  color  scales  the  intensity. 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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Transaural  Synthesis:  A  New  Technical 
Advance  in  Auditory  Measurement 

W.  Hartmann,  MSU. 


Scientific  Challenge: 

How  does  binaural  hearing 
disclose  the  locus  of  sound 
in  real  3D  environments? 

Transaural  Synthesis  Method: 

•  Eliminates  inter-aural  crosstalk,  compares 
localization  judgments  with  precisely  known 
acoustic  parameters  in  the  ear  canals. 

•  Tests  auditory  theory  by  manipulating 
Interaural  phase  and  amplitude,  comparing 
real  sounds  with  “virtual”  sounds. 

•  Tests  in  real  rooms  &  environments, 

with  varied  reverberation  -  no  headphones. 

•  Self-compensates  for  probe-tube  positions, 
enables  precise  reproducability. 


NOISE  SPECTRUM  MEASUREMENTS 
IN  EAR  CANAL 
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DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 


14 


Acoustic  Source  Separation: 

Recovering  Multiple  Speech  Signals  from  a  Single  Channel 


Research  Challenge: 

Devise  a  factorization  method  to: 

•  Recover  both  magnitude  arid  phase 
for  each  speech  component 
superimposed  in  a  single  channel 

•  Avoid  trial-and-error  basis  selection. 


Brian  King 
U.  Washington 
Ph.D.  Candidate 
& 

RH  Summer  Student 
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SPEECH  RECOVERY  INDEX 


Why  Phase  Matters: 
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New  Complex 
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New  Complex  Matrix  Factorization  method 
solves  superposition  and  phase  problems, 
eliminates  ad  hoc  basis  selection, 

&  Improves  speech  recognition  scores 


King  &  Atlas,  IEEE  Trans  on  Audio,  Speech  and  Language  Processing  (Nov.  2011) 
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How  do  Human  Listeners  Sort 
Overlapping  Speech  Information  ? 


Scientific  Question: 

When  each  ear  receives  a 
distinct  message,  can  the 
auditory  brain  extract  key 
information  from  both? 

Surprising  Result: 

Manipulation  of  word  predictability  in 
dichotic  listening  strongly  supports  a 
model  of  attention-switching,  rejects 
hypothesis  of  divided  attention. 


N.  Iyer,  B.  Simpson,  Principal  Investigators,  711  th  HPW/AFRL/RHCB  Battlespace  Acoustics 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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Coherent  Modulation  Analysis: 

Dr.  Les  Atlas,  University  of  Washington 


Mathematical  challenge: 

Estimate  and  separate  the  linear 
time-invariant  and  frequency-invariant 
components  of  an  acoustic  source. 

Approach: 

Theory  of  Complex  Modulation  Filtering, 

devised  by  Prof.  Les  Atlas  (originally  for 
analysis  of  speech  signals.) 


Unexpected  Application: 

A  simple,  one-microphone  set-up  detects  hole  quality 
while  drilling  in  airframe  composite  material. 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 


17 


Sensori-motor  Control  of  Natural 
Flight  and  Navigation 


Fundamental  Question: 

What  underlying  principles  drive 
biology’s  design  of  actuation  and 
sensing  architectures? 


Motivating  Observations 

from  Insect  Research  : 

•  Sensors  are  “noisy,”  redundant,  distributed 

in  non-orthogonal  coordinates. 

•  Inputs  fuse  across  modalities  prior  to 

activating  flight  muscles. 

•  No  conventional  distinctions  between 

estimate/control  or  inner/outer  loop 

•  Sensors  differ  radically  in  bandwidth  & 

temporal  response,  e.g.,  vision  lags 
mechanoreception. 


Wing  Steering 


Muscles 


Halteres 


Ocelli 


Campaniform 

Sensors 


Tangential 

Cells 


Descending 

Cells 


Haltere  Control 


Muscles 


Insect  Lab,  Eglin  AFB 
Dr.  Jennifer  Talley 


■h 
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Sensori-motor  Control  of  Natural 
Flight  and  Navigation 

▲  =  AFOSR  BIO-NAVIGATION  FUNDING  INITIATIVE 


T.  Daniel  (  U.  Washington): 

Wing  mechanosensor  functions. 

J.  Evers  (AFRL/RW): 

Natural  3D  flight  dynamics 

M.  Frye  (  UCLA): 

Higher-order  motion  detection 

S.  Humbert  (  U.  Maryland): 
Modeling  sensorimotor  control 

H.  Krapp  ( Imp.  College  London): 
Neural  basis  of  visual  steering 

P.  Krishnaprasad  (  U.  MD): 
Modeling  formation  flight  control 

S.  Reppert  (  U.  Mass): 
Clock-compensated  navigation 

R.  Ritzmann  (Case  Western): 
Adaptive  locomotion  control 


R.  Olberg  (  Union  College): 
Dragonfly  flight  to  target  capture 

S.  Sane  ( Tata  Institute): 

Insect  multisensory  integration 

P.  Shoemaker  (Tanner  Res.): 

Visual  detection  of  small  targets 

S.  Sterbing  (  U.  MD): 

Wing  sensors  in  bat  flight  control 

G.  Taylor  (Oxford): 

Raptor  pursuit  strategies  in  3D 

E.  Warrant  (Lund  U.): 

Nocturnal  navigation 

M.  Wehling  (AFRL/RW): 

Neural  analysis  of  optic  flow. 

M.  Willis  (Case  Western): 

Visual  /  olfactory  target  tracking 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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International  and  6.2 
Coordination 


W 


U.K. 


J.  Niven 


AFRL-DSTL  Working  Group 
Biologically-Motivated  Micro-Air- Vehicles 


“STATE  OF  THE  ART  REVIEW” 

Georgia  Tech 
15-18  June,  2010 


Organizers:  M.  Wehling,  AFRL.  P.  Biggins,  Dstl 
30  Participants  from  UK,  US,  Industry,  Academia,  &  Gov. 


Presentations: 

https://livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&obild=24091294 

&obiAction=browse&viewType=1 


M.  Willis 


S.  Humbert 


T.  Dani 


W  20 
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Vertebrates 


Invertebrates 


Hawks  &  Falcons 


Echolocating  Bats 


Megalopta  genalis 
(Nocturnal  Bee) 


Bombus  terrestris 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 


21 


Antennae  in  Flight  Control 

Dr.  Sanjay  Sane,  Tata  Institute,  Bangalore,  India 


Hawkmoth,  Daphnis  nerii 


•\  Precise  control  of  antennae  position 
is  critical  for  aerial  flight  maneuvers 

2  Control  system  is  closed-loop, 
with  sensory  latency  of  1 5  msec. 

3  Antennae  integrate  mechanosensory 
&  photoreceptor  inputs 


Antennal 
Sensori- Motor 
System 


Antennae  position 
in  flight  is  governed 
by  Bohm’s  bristles 
only  at  pedicel. 


AOARD  grant  via  T.  Erstfeld 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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Wings  are  Sensory  Receptors 


—  A  New  Research  Effort 


CORIOLIS 

MECHANICAL  REGIONS 


Hypothesis: 


Hawkmoth,  Manduca  sexta 


Wings  not  only  drive  flight,  but  also  detect  inertial  moments. 
-  -  -  strain  receptors  modulate  wing  shape  and  position. 


T.  Daniel  (biology)  and  K.  Morgensen  (Aeronautics)  U.  Washington. 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 
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The  Mode  Sensing  Hypothesis 

for  Biological  Flight  Control 


Proposed  by  3  AFOSR  scientists: 


Theory: 

Biological  sensors  and  actuators  compose 
a  suite  of  “matched  filters,”  tuned  to  salient 
patterns  (“modes”)  of  self-motion  - 
organized  to  combine  multiple  inputs  into 
actuation  signals  encoded  in  modal  coordinates. 


IN*# 


H.  Krapp 


MARYLAND 


IMPERIAL 

COLLEGE 

LONDON 


OXFORD 


Challenge  from  AFOSR: 

*  Test  the  theory  in  neurobiology  and  flight  behavior 

•  Devise  mathematical  emulation  suitable  for  MAVs 


H.G.  Krapp,  G.  K.  Taylor,  and  J.S 
Humbert  (in  press)  “The  Mode- 
Sensing  Hypothesis:  Matching 
Sensors,  Actuators  and  Flight 
Dynamics.” 
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10  vs 

cells 


Calliphora  vicina 
Lobular  Plate  Tangential  Cells 
(Compound  Eye  System) 


H.  Krapp,  Imperial  College,  London 


10  Vertical  System  Cells 

front 


VS9 
VS8 


rear 


•  Sensory  Integration  begins  by  selectively 
merging  these  inputs. 

•  The  resulting  “matched  filters”  for  self-motion 
are  robust  and  innate. 


*  V 
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V  J^Other  Motion  Sensor  Tuning  Di 

Compound  Eye  Tuning 


Plots  derived  from  single-unit  recordings 
during  controlled  rotational  stimulation 


Data  from  U.  Cambridge,  (M.  Parsons) 
&  Imperial  College,  London  (H.  Krapp) 
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Wing  Sensorimotor  Activation 

*•+  -  -  A  New  Program  Initiative 


Natural  Fliers’  Wing  Joints 

(  •  )  with  potential  to  be  controlled  during  flight 


Research  Questions: 

•  Discover  joint  coordination  timing 

•  Determine  functional  redundancy 

•  Do  joint  muscles  control  force  or  position? 

•  Discover  what  the  intrinsic  muscles  do. 


Bat  wing  has  Intrinsic, 
non-joint  muscles 


Plagiopatagiales  Muscles 
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Discovering  Bat  Wing  Musculature 
Dynamics  during  Flight 


Large  bat  (1 .2kg)  in  low  speed  flight  has  precise 
control  via  wing  skeletal  muscles  and  wing 
intramembranous  (intrinsic)  muscles 

ResearchTechniques: 

•  Electromyography  during  flight 

•  Thermal  videography  for  metabolic  load) 

•  Selective,  reversible  muscle  paralysis 

•  3D  X-ray  mapping  of  skeletal  motion 


Intrinsic  Muscles  Active 
only 


...A  Discovery  from  AFOSR  MURI 
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SUMMARY: 

Transformational  Impacts  &  Opportunities 


Hearing  protection: 

•  Massive  improvements  in  high-noise  attenuation. 

Advanced  auditory  modeling: 


•  Mathematics  for  coherent  modulation  analysis 

•  Neural- Inspired  analyses  to  parse  acoustic  scenes 

Optical  processing: 

•  Polarization  vision  and  signaling  adapted  from  biology 

•  Achromatic  1/4  wave  optical  retarders 

•  Emulating  compound  eye  in  new  optical  devices 


Autonomous  flight  control: 

•  Adaptive  airfoils  based  upon  bio-sensory  mechanisms 

•  Steering  based  upon  neural  autonomous  systems 

•  Discover  sensorimotor  basis  of  formation  flight 
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Questions? 


Thank  you  for  your  attention. 

Willard  Larkin,  Program  Manager,  AFOSR/NL 

703-696-7793 

Willard.Larkin@afosr.af.mil 
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Questions? 


Thank  you  for  your  attention. 

Willard  Larkin,  Program  Manager,  AFOSR/NL 

703-696-7793 

Willard.Larkin@afosr.af.mil 
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New  Lab  for  Insect  Vision 
Spectral  &  Polarization 
Electroretinography 
AFRL/RWG  PI:  Martin  Wehling 


Robber  Fly 


No  Ocelli 
Halteres 
No  VS  cells 
5-18  HS  cells 
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Dynamics  of  Bat  Wing  Musculature 

S.  Swartz,  T.  Roberts,  Brown  Univ.,  2011 
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Large  bat  (1 .2kg)  in  low  speed  flight  maintains 
precise  control  via  wing  skeletal  muscles  and  wing 
intramembranous  muscles 


Techniques: 

Electromyography  during  flight 
Thermal  videography  (to  measure  metabolic  load) 
Selective,  reversible  muscle  paralysis 
3D  X-ray  mapping  of  skeletal  motion 
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1 .5  gm  EMG  Telemetry 
for  in-flight  muscle  activity 


Muscles  active  during  upstroke: 
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Time  (s) 


0.2  0.25  0.3 

Time  (s) 


DISTRIBUTION  A:  Approved  for  public  release;  distribution  is  unlimited. 


36 


